Method and apparatus for distributed analyses of images

ABSTRACT

A method and apparatus for intelligent distributed analyses of images including capturing the images and analyzing the captured images, where feature information is extracted from the captured images. The extracted feature information is used in determining whether a predefined condition is met, and the extracted feature information is transmitted for further analysis when the predefined condition is met. The extracted feature information is stored and is used to generate statistical information related to the extracted feature information. Further, additional feature information is provided from other databases to implement further analysis including an event detection or recognition. Accordingly, distributed intelligent analyses of images is provided for analyzing captured images to efficiently and effectively implement event detection or recognition.

BACKGROUND OF THE INVENTION Description of the Related Art

As the use of images for event detection or recognition becomes morepervasive, efficient ways of analyzing images becomes essential.Generally, analyzing images for event detection or recognition consistsof capturing the images and analyzing the captured images using humanoperators. For example, images of an entrance into a building or roomcaptured by a video camera are maintained such that a human operatorsubsequently reviews the captured images to determine whether a personhas entered the building or room. In this situation, the human operatormust constantly monitor the captured images to determine an occurrenceof an event or an existence of a condition, and a significant amount ofcaptured images that need to be monitored must be transmitted from thevideo camera.

A typical system for capturing and analyzing images requires that imagesignals be carried over dedicated coaxial cable, fiber optic line, etc.,and further requires that electrical power be supplied to supportcontinuous operation. Thus, the cost of the typical system issignificant.

Other solutions for image processing have been proposed where processingof captured images is implemented using a processor in an imagecapturing device such that an alert or an alarm is triggered when achange occurs. For example, an image capturing device, such a camera infront of a store, may be provided with a processor for processing imagescaptured by the camera so that the camera triggers an alarm when anumber of pixels between consecutive images exceeds a certain threshold.However, a processor installed on a camera has limited capability due tosize, weight, cost, power limitations, etc., and thus, does not enablecomplex event detection or recognition. Further, due to the limitedprocessing capability of the processor installed on the camera, accurateevent detection or recognition can not be implemented, therebyincreasing the rate of false alerts or alarms.

Accordingly, it is important to provide intelligent distributed analysesof images for efficient event detection or recognition. This becomesespecially important as image analysis continues to be necessitated bydifferent purposes, such as for security purposes, etc. Thus, there is aneed for intelligent distributed analyses of images that addresses theabove-mentioned and other limitations.

SUMMARY OF THE INVENTION

Various embodiments of the present invention provide a method including(a) capturing digital image data by a sensor, (b) extracting featureinformation in real-time from the captured digital image data anddetermining whether the extracted feature information meets a predefinedcondition by the sensor, and (c) transmitting the extracted featureinformation to a remote device for further analysis when the sensordetermines that that the extracted feature information meets thepredefined condition.

Various embodiments of the present invention provide an apparatusincluding (a) a sensor capturing images, and (b) a processor extractingfeature information from the captured images, determining whetherchanges in the extracted feature information pass a threshold level andcausing the extracted information to be sent to a remote device forfurther analysis when the processor determines that the changes pass thethreshold level.

Moreover, various embodiments of the present invention provide adistributed event detection apparatus including (a) a plurality of imagesensors capturing image data and interpreting the captured image data toextract feature information from the captured image data, and (b) aserver connected with the plurality of image sensors receiving theextracted feature information for further analysis including an eventdetection, where the plurality of image sensors transmit the extractedfeature information of the captured image to the server when theextracted feature information meets a predefined condition.

Various embodiments of the present invention provide present inventionfurther provides a method including (a) capturing a plurality of lowresolution digital images via multiple distributed sensors, (b)combining the captured plurality of low resolution digital images into asingle high resolution digital image, and (c) extracting featureinformation from the high resolution digital image.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects and advantages of the invention will becomeapparent and more readily appreciated from the following description ofthe embodiments, taken in conjunction with the accompanying drawings ofwhich:

FIG. 1 is a diagram illustrating a process of capturing images,extracting feature information from the captured images, determiningwhether the extracted feature information meets a predefined conditionand transmitting the extracted feature information for further analysis,according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating an apparatus for capturing images,interpreting the captured images to extract feature information from thecaptured images and sending the extracted information for furtheranalysis, according to an embodiment of the present invention.

FIG. 3 is a diagram illustrating a process for multiple levels of eventdetection or recognition, according to an embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference will now be made in detail to the present embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to the like elementsthroughout. The embodiments are described below to explain the presentinvention by referring to the figures.

FIG. 1 is a diagram illustrating a process 100 for capturing images,extracting feature information from the captured images, determiningwhether the extracted feature information meets a predefined conditionand transmitting the extracted feature information from the capturedimages for further analysis, according to an embodiment of the presentinvention. Referring to FIG. 1, in operation 10, images are captured.For example, according to an embodiment of the present invention, stillimages are captured using a sensor. The present invention, however, isnot limited to using one sensor. Instead, multiple sensors may beprovided to capture still images. Moreover, the present invention is notlimited to the captured images being “still” images.

It is well known how to capture an image. The image may be capturedusing a sensor such as, for example, a digital camcorder, a digitalstill camera, a cellular telephone, a personal digital assistant, orother devices responsive to a particular motion, sound, light, etc.Generally, these sensors incorporate an image capturing device,typically a charge coupled device (CCD) or a CMOS image sensor (CIS),that registers the image, and image processing hardware and/or softwarethat converts a raw signal produced by the image sensor to useful imagedata, such as CCIR or JPEG data. However, the present invention is notlimited to any particular type of sensor.

Referring to FIG. 1, from operation 10, the process 100 moves tooperation 12, where feature information is extracted from the capturedimages. The feature information can be extracted using various methods,and thus, the present invention is not limited to any specific method ofextracting feature information from captured images. For example, thepresent invention can be set up to extract essential feature informationrelated to relative brightness or chromaticity values of images todetect a change between the brightness or chromaticity values of images.Further, an algorithm can be used to determine when a change hasoccurred, for example, by computing a difference between consecutiveimages based on a count of a number of pixels in the images andindicating when the difference exceeds a certain value. Various featureinformation extraction methods are well known.

From operation 12, the process 100 moves to operation 14, where it isdetermined whether the extracted feature information meets a predefinedcondition. For example, according to an embodiment of the presentinvention, it is determined whether a change has occurred based on adifference computed between a number of pixels in consecutive imagesusing an algorithm. In this situation, the change based on thedifference computed between the number of pixels in the images is thepredefined condition. However, the present invention is not limited to apredefined condition determined based on a difference between computednumber of pixels.

From operation 14, the process 100 moves to operation 16, where theextracted feature information is transmitted for further analysis whenthe extracted feature information meets the predefined condition. Forexample, in relation to the example discussed in the previous paragraph,the extracted information is transmitted for further analysis upondetermining that the change has occurred based on the differencecomputed between the number of pixels in the images. Further, accordingto an embodiment of the present invention, the transmitted featureinformation includes, for example, a single image or a small number ofimages.

The present invention is not limited to a particular number ofsubsequent analyses of feature information. For example, upondetermining that the change has occurred based on the differencecomputed between the number of pixels in the images and transmitting thefeature information, it is possible to further transmit the featureinformation for other subsequent analysis.

Accordingly, in process 100, the feature information is extracted fromthe captured images and it is determined whether the extracted featureinformation meets a predefined condition. The extracted featureinformation is then transmitted for further analysis when the extractedfeature information meets the predefined condition, thereby implementingintelligent distributed analyses of captured images.

FIG. 2 is a diagram illustrating an apparatus for capturing images,interpreting the captured images to extract feature information from thecaptured images and sending the extracted information for furtheranalysis, according to an embodiment of the present invention. Theapparatus in FIG. 2 can be used to implement the process 100 of FIG. 1.

Referring to FIG. 2, a distributed event detection apparatus 200 isprovided. The distributed event detection apparatus 200 comprisesmultiple sensors 40 to capture still images. The distributed eventdetection apparatus 200 also includes a hub 80. The sensors 40 areconnected with the hub 80. The hub 80 is connected with a remote device130 by a connection 120.

The embodiment in FIG. 2 shows an example in which some of the sensors40 are connected with the hub 80 by wireless connections 60 and 60 acommunicating with a wireless interface 70 of the hub 80 via wirelessinterfaces 70 a and 70 b. The wireless connections 60 and 60 a can beenabled using wireless communication protocols, such as IEEE 802.11b,Bluetooth, etc. However, the wireless connections 60 and 60 a are notlimited to a specific wireless communication protocol.

The embodiment in FIG. 2 shows other of the sensors 40 connected withthe hub 80 by wired connections 50 and 50 a.

Although FIG. 2 shows some of the sensors 40 connected with the hub 80by wireless connections 60 and 60 a and others of the sensors 40connected with the hub 80 by wired connections 50 and 50 a, embodimentsof the present invention are not limited to the use of wireless and/orwired connections, or any particular types of protocols. Moreover,embodiment of the present invention are not limited to different sensorshaving different types of connections. In addition, embodiments of thepresent invention are not limited to using any particular number ofsensors 40.

As mentioned above, the apparatus in FIG. 2 can be used to implement theprocess in FIG. 1. For example, in an embodiment of the presentinvention, operations 10 and 12 of FIG. 1 can be implemented by sensors40, with operation 14 and 16 in FIG. 1 being implemented by the hub 80.

As a more concrete example, referring to FIG. 2, the sensors 40 capturestill images, extract feature information from the captured images andtransmit the extracted feature information to the hub 80. For example,according to an embodiment of the present invention, the sensors 40extract feature information related to chromaticity values of thecaptured images based on an algorithm and transmit the extracted featureinformation to the hub 80 for determination of whether the chromaticityvalues meet a predefined condition. However, there are many differentmanners and algorithms for extracting feature information, and thepresent invention is not limited to any particular manner or algorithm.Therefore, in this example, the sensors 40 perform operations 10 and 12in FIG. 1, and the hub 80 performs operation 14. The hub 80 might alsotransmit the feature information to the remote device 130 for furtheranalysis, thereby performing operation 16. In other embodiments, the hub80 may also perform the further analysis.

In a different embodiment of the present invention, operation 10 in FIG.1 can be implemented by a sensor 40, operation 12 in FIG. 1 can beimplemented by the hub 80, and operations 14 and 16 in FIG. 1 can beimplemented by the remote device 130.

As a more concrete example, referring to FIG. 2, the sensors 40 capturestill images, and transmit the captured images to the hub 80. Forexample, according to an embodiment of the present invention, thesensors 40 capture images and transmit the captured images to the hub 80where the hub extracts feature information related to, for example,chromaticity values of the captured images and transmits the featureinformation to the remote device 130. The remote device 130 determineswhether the chromaticity values meet a predefined condition. Further,the remote device 130 can transmit the feature information for furtheranalysis upon determining that the chromaticity values meet thepredefined condition. Therefore, in this example, the sensors 40 performoperations 10 in FIG. 1, the hub 80 performs operation 12 and the remotedevice 130 performs operations 14 and 16.

Further, in some embodiments, operations 10, 12, 14 and 16 of FIG. 1 canperformed by the sensors 40.

As a more concrete example, referring to FIG. 2, the sensors capture theimages, extract feature information from the captured images anddetermine whether the feature information meets a certain condition. Inthis embodiment, the sensors 40 perform operations 10, 12, 14 and 16 ofFIG. 1, and the hub 80 performs the further analysis. For example, oneof the sensors 40 shown in FIG. 2 that is provided with a processor 42determines whether the chromaticity values of captured images exceeds athreshold value and transmits the feature information related to thechromaticity values of the images to the hub 80 for further analysis.Accordingly, the threshold that has been determined to be met by one ofthe sensors 40 can be re-evaluated by the hub 80. However, the presentinvention is not limited to implementing a determination of anoccurrence of a condition at a sensor or a hub, instead, suchdetermination can be implemented at a sensor and/or a hub. While onlyone of the sensors 40 in FIG. 2 is shown to have a processor 42, thepresent invention is not limited to any particular number of processorsthat are provided to the sensors 40. For example, it is possible toprovide a processor for each of the sensors 40 that are connected withthe hub 80. Further, the present invention is not limited to anyspecific number of sensors.

The present invention is not limited to operations 10 through 16 of FIG.1 being perform by a specific combination of the sensors 40, the hub 80and the remote device 130. Instead, the operations 10 through 16 of FIG.1 can be variously implemented using the sensors 40, the hub 80 and theremote device 130.

FIG. 3 is a diagram illustrating a process 300 for multiple levels ofevent detection or recognition, according to an embodiment of thepresent invention. As shown in FIG. 3, the process 300 includes multiplelevels of detection 150, 152 and 154. Specifically, FIG. 3 shows a firstlevel of detection 150, a second level of detection 152 and a thirdlevel of detection 154 of the process 300 for multiple levels ofdetection. The first level of detection 150 receives and processes thereceived images 140 to determine an existence of a condition or an eventand transmits a signal 142 to the second level of detection 152 forfurther analysis. For example, the sensors 40 shown in FIG. 2 captureimages of an entrance into a room (images 140 in FIG. 3) and extractfeature information related to a number of pixels in the captured imagesand determine that there is a difference between the number of pixels inthe images at the first level of detection 150 of FIG. 3. While thefirst level of detection 150 is described in relation to a differencebetween a number of pixels, a first level of detection according to thepresent invention is not limited to any specific type of detection ordetermination.

Further, the second level of detection 152 receives and further analyzesthe signal 142 to determine an existence of a condition or an event andtransmits a signal 144 to the third level of detection 154. Accordingly,in relation to the example in the previous paragraph, the sensors 40 (inFIG. 2) transmit extracted feature information related to the number ofpixels in the captured images 142 to a hub 80 (in FIG. 2) for a secondlevel of detection 152 of FIG. 3. In this instance, the hub 80implements a complex algorithm to determine whether the difference inthe number of pixels determined by the sensors 40 at the first level ofdetection 150 indicates whether a person has entered the room, asopposed to another object, in the second level of detection 152.Further, accordingly to an embodiment of the present invention, the hub80 transmits signal 144 indicating a determination of whether a personhas entered the room for further analysis to, for example, a remotedevice 130 shown in FIG. 2

The third level of detection 154 receives and further analyzes thesignal 144 to further determine or confirm the existence of thecondition or the event, and transmits a signal 146 to trigger an alarmor a notification. For example, in relation to the example in twoprevious paragraphs, the remote device 130 shown in FIG. 2 receives thesignal 144 indicating that a person has entered the room from the hub 80(also shown in FIG. 2) and makes a further determination, such aswhether an identity of the person determined to have entered the room bythe hub 80 matches a predetermined identity. According to an embodimentof the present invention, the remote device 130 also transmits thesignal 146 to trigger an alarm. This enables multiple levels of eventdetection or recognition. While the above descriptions of the multiplelevels of detection are discussed using a single method of featureextraction (i.e. chrominance changes or pixel number changes betweenimages), the present invention is not limited to using a single methodof feature extraction. For example, both the of chrominance or pixelnumber changes between the images can be implemented to determinewhether image data needs to be transmitted for further analysis.

Accordingly, a multi-level event detection or recognition isimplemented, according to an aspect of the present invention. Forexample, an image sensor, a hub and a remote device may be provided toimplement first, second and third levels of detection to execute themulti-level event detection or recognition similar to the illustrationin FIG. 3. While the process 300 for multiple levels of event detectionor recognition in FIG. 3 is illustrated using three levels of detection150, 152 and 154, the present invention is not limited to three levelsof detection. For example, two levels of detection can be implementedwhere a sensor captures images of an entrance into a room and extractsfeature information related to a number of pixels in the captured imagesfor determining whether there is a difference between the number ofpixels in the images indicating an entrance into the room at a firstlevel of detection, and the sensor can transmit a result of thedetermination to a hub or a remote device for a second level ofdetermination, where the hub or remote device triggers an alarm based onwhether the second level of determination indicates that the differencebetween the number of pixels in the images by the sensor results in adetermination that a person has entered the room. Accordingly, thepresent invention can be implemented using two levels of event detectionor recognition.

Accordingly, a method and apparatus for intelligent distributed analysesof images is provided. This enables interpretation of captured images atmultiple levels, thereby providing an efficient and effective method andapparatus for image analyses. The multiple levels of interpretationenable ambiguities that may exist when analyzing the captured images tobe reduced. Further, the flow of information related to the capturedimages is reduced as the information is transmitted to the multiplelevels.

The present invention also enables extraction of feature informationfrom captured images, analysis of the feature information in real-timeand transmission of the analyzed feature information for furtheranalysis. This allows the essence of the feature information to beextracted from the captured images and be sent upstream for furtheranalysis, thereby reducing the flow of information. Further, accordingto an embodiment of the present invention, the sensors extract featureinformation from captured images, thereby allowing real-time review ofthe captured images.

Further, the present invention provides distributed analyses of imagesthat can be implemented using various different known feature extractionmethods. This allows the distributed analyses of images of the presentinvention to be used with known methods of extracting meaningful featureinformation from images.

Additionally, the present invention provides an apparatus including asensor to capture images and a processor to extract feature informationfrom the captured images. The processor enables the determination ofwhether changes in the extracted feature information pass a thresholdlevel, and causes the extracted information to be sent to a remotedevice for further analysis when the processor determines that thechanges pass the threshold level.

The present invention are also provides a method of capturing aplurality of low resolution digital images via multiple distributedsensors and combining the captured plurality of low resolution digitalimages into a single high resolution digital image. The method alsoincludes extracting feature information from the high resolution digitalimage, determining whether the extracted feature information meet apredefined condition and transmitting the extracted feature informationfor an event detection upon determining that the low resolution digitalimages meet the predefined condition.

Although a few embodiments of the present invention have been shown anddescribed, it will be appreciated by those skilled in the art thatchanges may be made in these embodiments without departing from theprinciples and spirit of the invention, the scope of which is defined inthe appended claims and their equivalents.

1. A method, comprising: capturing digital image data by a sensor;extracting feature information in real-time from the captured digitalimage data and determining whether the extracted feature informationmeets a predefined condition by the sensor; and transmitting theextracted feature information to a remote device for further analysiswhen the sensor determines the extracted feature information meets thepredefined condition; wherein the remote device executes the furtheranalysis in accordance with data received from a database, and the datareceived from the database is combined to provide a high resolutionimage.
 2. The method according to claim 1, wherein the digital imagedata is periodically captured by the sensors.
 3. The method according toclaim 1, wherein the captured digital image data are still images. 4.The method according to claim 1, wherein the extracted featureinformation is transmitted to the remote device for an event detection.5. The method according to claim 1, wherein the extracted featureinformation is stored and the further analysis by the remote device isexecuted in accordance with the stored feature information.
 6. Anapparatus, comprising: a sensor capturing images; and a processorextracting feature information from the captured images, determiningwhether changes in the extracted feature information pass a thresholdlevel, and causing the extracted information to be sent to a remotedevice for further analysis when the processor determines that thechanges pass the threshold level, wherein passing the threshold levelindicates that an unidentified person has entered a room, and thefurther analysis determines whether the person matches a predeterminedidentity.
 7. The apparatus according to claim 6, further comprising: ahub connected with the sensor, wherein the processor is located in thehub, and the remote device is remote from the sensor and remote from thehub.
 8. The apparatus according to claim 7, wherein the hub includes adatabase, and the extracted feature information is stored in thedatabase to generate statistical information related to the featureinformation.
 9. The apparatus according to claim 6, wherein theprocessor is located locally with the sensor.
 10. The apparatusaccording to claim 7, wherein the captured images are still images. 11.A distributed event detection apparatus, comprising: a plurality ofimage sensors capturing image data and interpreting the captured imagedata to extract first feature information from the captured image data;and a server connected with the plurality of image sensors receiving theextracted first feature information for further analysis, wherein theplurality of image sensors transmit the extracted first featureinformation of the captured image to the server when the extracted firstfeature information meets a first predefined condition, and the servertransmits extracted second feature information, when the furtheranalysis meets a second predefined condition, and a remote devicereceives the extracted second feature information for still furtheranalysis, including an event detection, when the still further analysismeets a third predefined condition.
 12. The distributed event detectionapparatus according to claim 11, wherein the plurality of image sensorsinterpret the captured image data using multiple algorithms to extractthe feature information from the image data.
 13. The distributed eventdetection apparatus according to claim 11, wherein the plurality ofimage sensors periodically transmit an entire portion of the capturedimage data to the server for an event detection.
 14. The distributedevent detection apparatus according to claim 11, wherein the capturedimage data has a size less than the extracted information of thecaptured image data.
 15. The distributed event detection apparatusaccording to claim 11, wherein the server stores the extracted featureinformation, and generates statistical information related to theextracted feature information.
 16. The distributed event detectionapparatus according to claim 11, wherein the captured image data arestill images.
 17. The distributed event detection apparatus according toclaim 11, wherein additional feature information is provided to theserver from other databases for the further analysis including an eventdetection by the server.
 18. A method, comprising: capturing a pluralityof low resolution digital images via multiple distributed sensors;combining the captured plurality of low resolution digital images into asingle high resolution digital image; and extracting feature informationfrom the high resolution digital image, determining whether theextracted feature information meet a predefined condition andtransmitting the extracted feature information for an event detectionupon determining that the low resolution digital images meet thepredefined condition.